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Scientific  and  Technical  Objectives 

This  MURI  project  develops  a  SMartVISION  general-purpose,  autonomous  neural 
system  for  vision,  object  recognition,  and  tracking  applications,  to  explain  how  the  brain 
sees,  and  to  transfer  new  computer  vision  architectures  into  technology.  SMartVISION 
predictions  have  been  successfully  tested  in  psychophysics  and  neurobiology  labs. 
SMartVISION  embodies  a  revolutionary  paradigm  shift  in  intelligent  computation.  It  is 
designed  to  operate  in  real-time  within  noisy  environments  for  which  rules  are  not 
known,  and  which  contain  rare  but  important  events,  unexpected  events,  incomplete 
data,  irregular  statistical  drifts,  and  different  amounts  of  morphological  variability  in 
objects  to  be  detected  and  recognized.  A  companion  architecture  for  distributed 
planning,  decision,  and  action  helps  to  actively  acquire  information  by  interacting  with 
the  SMartVISION  system,  including  circuits  to  control  attention  shifts  and  eye 
movements  to  fixate  regions  of  interest.  SMartVISION  is  quantitatively  simulating  the 
dynamics  of  identified  nerve  cells,  in  known  anatomical  circuits,  and  the  emergent 
behaviors  that  they  control.  It  embodies  qualitatively  new  computational  paradigms  with 
novel  concepts  and  mechanisms  for  intelligent  computing  in  response  to  a  rapidly 
changing  world,  notably  the  paradigms  of  Laminar  Computing  and  of  Complementary 
Computing.  These  concepts  and  mechanisms  have  begun  to  break  through  old  barriers 
in  technology. 
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Approach 

Senior  faculty  coordinate  small  teams  of  faculty/student  researchers  on  projects  to 
develop  the  unified  SMartVISION  system.  Biological  modeling  projects  discover 
fundamental  organizational  principles  and  mechanisms  that  enable  the  brain  to  adapt  in 
real-time  to  unexpected  environmental  challenges,  translate  them  into  mathematical 
models,  and  use  the  models  to  quantitatively  simulate  large  brain  and  behavioral  data 
bases  about  vision,  object  recognition,  and  tracking.  These  models  have  introduced  two 
revolutionary  new  paradigms  into  intelligent  computing,  Laminar  Computing  and 
Complementary  Computing,  whose  impact  will  be  increasingly  felt  during  the  next 
several  decades.  A  linkage  between  brain  and  behavior  is  necessary  for  technology 
transfer,  because  brain  mechanisms  say  how  it  works,  and  behavioral  functions  say 
what  it  is  for.  Moreover,  models  that  can  adapt  autonomously  in  real  time  to  a  changing 
world  are  of  great  importance  in  solving  outstanding  technological  problems.  That  is 
why,  on  the  technological  side,  brain/behavior  models  from  BU  have  been  used  and 
further  developed  by  a  number  of  companies,  hospitals,  and  national  labs  to  process 
data  from  artificial  sensors  such  as  synthetic  aperture  radar,  laser  radar,  multispectral 
infrared,  night  vision,  nuclear  magnetic  resonance,  and  high  altitude  photography  for 
large-scale  applications  to  DoD  applications  and  technology. 

Concise  Accomplishments 

Biological  projects  include  laminar  cortical  and  subcortical  models  of:  (1)  3D  vision  and 
figure-ground  separation  in  response  to  natural  scenes  and  psychophysical  displays;  (2) 
3D  form-motion  interactions  to  generate  representations  of  object  direction  and  speed; 
(3)  3D  shape-from-texture;  (4)  learning  and  recognition  of  object  textures  using  surface- 
based  spatial  attention  and  object  attention;  (5)  learning  view-invariant  3D  object 
categories  from  multiple  2D  view  categories  using  active  eye  movements,  surface- 
based  spatial  attention,  and  object  attention;  (6)  learning  to  recognize  natural  scenes 
using  multiple-scale  filters  and  form-fitting  spatial  attention;  (7)  temporary  storage  of 
event  sequences  in  working  memory,  learning  of  sequential  plans,  and  sequence 
performance  during  cognitive  information  processing;  (8)  coordinated  ballistic  and 
smooth  pursuit  predictive  movements  that  maximize  visual  acuity;  (9)  navigation  and 
obstacle  avoidance  using  optic  flow  while  eyes/cameras  scan  the  environment;  (10) 
applications  of  biological  models  of  normal  cognition  to  understand  autism. 
Technological  projects  include:  (1)  an  image  processing  system  to  generate  visible 
chromatic  and  achromatic  scenes  under  variation  illumination  conditions;  (2)  information 
fusion  by  unsupervised  discovery  of  rules  and  hierarchical  knowledge  from  multi-faceted 
data;  (3)  image  analysis  by  long-range  object  completion  and  figure-ground 
segmentation  to  locate  watersheds,  fault  lines,  and  roads;  (4)  a  color  vision  model  that 
directs  attention  to  small  objects  in  complex  scenes;  (5)  a  testbed  to  predict  HIV 
resistance  to  antiretroviral  therapy;  (6)  a  technology  website  and  image  analysis  and 
classification  toolkits;  (7)  3D  VLSI  CMOS  implementation  of  neocortical  processing;  (8) 
silicon  neurons  and  photoreceptors  in  3D  CMOS  technology;  (9)  wireless  architecture 
for  cortical  distributed  processing. 

Expanded  Accomplishments 

Selected  accomplishments  are  summarized  here: 


View-invariant  object  learning  and  recognition:  Often  objects  can  be  recognized 
from  different  viewpoints.  How  can  the  brain,  or  a  computer  vision  system,  accomplish 
autonomous  learning  and  recognition  of  an  object  from  multiple  viewpoints  while 
scanning  a  scene  with  eye  movements?  How  does  the  brain  avoid  the  problem  of 
erroneously  classifying  parts  of  different  objects  together?  How  are  attention  and  eye 
movements  intelligently  coordinated  to  facilitate  object  learning?  A  neural  model 
provides  a  unified  mechanistic  explanation  of  how  spatial  and  object  attention  work 
together  to  search  a  scene  and  learn  what  is  in  it.  The  ARTSCAN  model  of  Fazl, 
Grossberg,  and  Mingolla  predicts  how  an  object’s  surface  representation  generates  a 
form-fitting  distribution  of  spatial  attention,  or  “attentional  shroud.”  All  surface 
representations  dynamically  compete  for  spatial  attention  to  form  a  shroud.  The  winning 
shroud  persists  during  active  scanning  of  the  object.  The  shroud  maintains  sustained 
activity  of  an  emerging  view-invariant  category  representation  while  multiple  view- 
specific  category  representations  are  learned  and  are  linked  through  associative 
learning  to  the  view-invariant  object  category.  The  shroud  also  helps  to  restrict  scanning 
eye  movements  to  salient  features  on  the  attended  object.  Object  attention  plays  a  role 
in  controlling  and  stabilizing  the  learning  of  view-specific  object  categories.  Spatial 
attention  hereby  coordinates  the  deployment  of  object  attention  during  object  category 
learning.  Shroud  collapse  releases  a  reset  signal  that  inhibits  the  active  view-invariant 
category  in  the  What  cortical  processing  stream.  Then  a  new  shroud,  corresponding  to 
a  different  object,  forms  in  the  Where  cortical  processing  stream,  and  search  using 
attention  shifts  and  eye  movements  continues  to  learn  new  objects  throughout  a  scene. 
The  model  mechanistically  clarifies  basic  properties  of  attention  shifts  (engage,  move, 
disengage)  and  inhibition  of  return.  It  simulates  human  reaction  time  data  about  object- 
based  spatial  attention  shifts,  and  learns  with  98.1%  accuracy  and  a  compression  of 
430  on  a  letter  database  whose  letters  vary  in  size,  position,  and  orientation. 

Texture  segregation  by  visual  cortex:  A  neural  model  called  dARTEX  was  developed 
by  Bhatt,  Carpenter,  and  Grossberg  to  demonstrate  how  laminar  interactions  in  the 
visual  cortex  may  learn  and  recognize  object  texture  and  form  boundaries.  The  model 
unifies  five  interacting  processes:  region-based  texture  classification,  contour-based 
boundary  grouping,  surface  filling-in,  spatial  attention,  and  object  attention.  The  model 
shows  how  form  boundaries  can  determine  regions  in  which  surface  filling-in  occurs; 
how  surface  filling-in  interacts  with  spatial  attention  to  generate  a  form-fitting  distribution 
of  spatial  attention,  or  attentional  shroud;  how  the  strongest  shroud  can  inhibit  weaker 
shrouds;  and  how  the  winning  shroud  regulates  learning  of  texture  categories,  and  thus 
the  allocation  of  object  attention.  The  model  can  discriminate  abutted  textures  with 
blurred  boundaries  and  is  sensitive  to  texture  boundary  attributes  like  discontinuities  in 
orientation  and  texture  flow  curvature  as  well  as  to  relative  orientations  of  texture 
elements.  The  model  quantitatively  fits  the  Ben-Shahar  and  Zucker  (2004)  human 
psychophysical  data  on  orientation-based  textures.  Surface-based  attentional  shrouds 
improve  texture  learning  and  classification:  Brodatz  texture  classification  rate  varies 
from  95.1%  to  98.6%  with  correct  attention,  and  from  74.1%  to  75.5%  without  attention. 
Object  boundary  output  of  the  model  in  response  to  photographic  images  was  favorably 
compared  to  computer  vision  algorithms  and  human  segmentations. 


Scene  understanding:  This  project  addresses  the  problem  of  how  humans  rapidly 
recognize  a  scene.  It  clarifies  how  neural  models  can  capture  this  biological 
competence  to  achieve  state-of-the-art  scene  classification.  The  ARTSCENE  model  of 
Grossberg  and  Huang  classifies  natural  scene  photographs  by  using  multiple  spatial 
scales  to  efficiently  accumulate  evidence  for  gist  and  texture.  ARTSCENE  embodies  a 
coarse-to-fine  Texture  Size  Ranking  Principle  whereby  spatial  attention  processes 
multiple  scales  of  scenic  information,  ranging  from  global  gist  to  local  properties  of 
textures.  The  model  can  incrementally  learn  and  predict  scene  identity  by  gist 
information  alone  and  can  improve  performance  through  selective  attention  to  scenic 
textures  of  progressively  smaller  size.  ARTSCENE  discriminates  4  landscape  scene 
categories  (coast,  forest,  mountain  and  countryside)  with  up  to  91.58%  correct  on  a  test 
set,  outperforms  alternative  models  in  the  literature  which  use  biologically  implausible 
computations,  and  outperforms  component  systems  that  use  either  gist  or  texture 
information  alone.  Model  simulations  also  show  that  adjacent  textures  form  higher-order 
features  that  are  also  informative  for  scene  recognition. 

Thus,  ARTSCENE  is  part  of  a  larger  research  program,  illustrated  as  well  by  the  above 
three  projects,  for  clarifying  how  processes  of  multiple-scale  texture  filtering  and 
grouping,  spatial  attention,  fast  category  learning  and  recognition,  and  scanning  eye 
movements  can  work  together  to  achieve  impressive  visually-based  learning  and 
recognition  competences. 

3D  shape-from-texture:  In  addition  to  its  value  in  the  recognition  of  natural  objects, 
texture  information  may  be  used  to  generate  representations  of  object  shape  that 
cannot  be  easily  explained  using  more  traditional  3D  modeling  approaches.  To 
complement  the  analysis  of  how  multiple-scale  filters  are  used  for  texture  recognition,  a 
LIGHTSHAFT  (LIGHTness-and-SHApe-From-Texture)  neural  model  was  developed  by 
Grossberg,  Kuhlmann,  and  Mingolla  to  clarify  how  cortical  areas  VI,  V2,  and  V4  interact 
to  convert  a  textured  2D  image  into  a  representation  of  curved  3D  shape.  Two  basic 
problems  were  solved  to  achieve  this:  (1)  Patterns  of  spatially  discrete  2D  texture 
elements  were  transformed  into  a  spatially  smooth  surface  representation  of  3D  shape. 
(2)  Changes  in  the  statistical  properties  of  texture  elements  across  space  induced  the 
perceived  3D  shape  of  this  surface  representation.  This  is  achieved  in  the  model 
through  multiple-scale  filtering  of  a  2D  image,  followed  by  a  cooperative-competitive 
grouping  network  that  coherently  binds  texture  elements  into  boundary  webs  at  the 
appropriate  depths  using  a  scale-to-depth  map  and  a  subsequent  depth  competition 
stage.  These  boundary  webs  then  gate  filling-in  of  surface  lightness  signals  in  order  to 
form  a  smooth  3D  surface  percept.  The  model  quantitatively  simulates  challenging 
psychophysical  data  about  perception  of  prolate  ellipsoids  (Todd  and  Akerstrom,  1987, 
J.  Exp.  Psych.,  13,  242).  In  particular,  the  model  represents  a  high  degree  of  3D 
curvature  for  a  certain  class  of  images,  all  of  whose  texture  elements  have  the  same 
degree  of  optical  compression,  in  accordance  with  percepts  of  human  observers. 
Simulations  of  3D  percepts  of  an  elliptical  cylinder,  a  slanted  plane,  and  a  photo  of  a 
golf  ball  are  also  presented. 


Information  fusion:  Unsupervised  discovery  of  rules  and  hierarchical  knowledge  from 
multi-faceted  data:  Carpenter,  Amis,  Ogas,  and  Olivera  developed  new  methods  that 
build  upon  a  novel  approach,  which  was  introduced  in  a  prior  MURI  project,  to  the 
information  fusion  problem.  These  methods  derive  consistent  knowledge  from  sources 
that  are  paradoxically  both  inconsistent  and  accurate.  A  new  ARTMAP  neural  network 
system  derives  hierarchical  knowledge  structures  from  nominally  inconsistent  training 
data.  The  system  learns,  for  example,  that  disparate  pixels  map  to  the  output  class 
"beach";  but,  if  similar  or  identical  pixels  are,  at  other  times,  labeled  "plage"  or  "open 
space"  or  "natural",  the  system  learns  to  associate  multiple  classes  with  a  given  input. 
Testbed  image  examples  have  shown  that  the  overall  pattern  of  distributed  predictions 
can  reveal  a  knowledge  hierarchy  which  guides  the  production  of  consistently  layered 
maps  of  test  regions.  Even  though  no  inter-class  relationships  are  specified  during 
training,  the  system  uses  distributed  activation  patterns  of  learned  codes  to  derive 
knowledge  of  relationship  rules,  confidence  estimates,  equivalence  classes,  and 
hierarchical  structures. 

Image  analysis:  Long-range  object  completion  and  figure-ground  segmentation:  The 
CONFIGR  (CONtour  Figure  GRound)  model  of  Carpenter,  Mingolla,  and  Gaddam  is  a 
computational  model  based  on  principles  of  biological  vision  that  completes  sparse  and 
noisy  image  figures.  Within  an  integrated  vision/recognition  system,  CONFIGR  posits  an 
initial  recognition  stage  which  identifies  figure  pixels  from  spatially  local  input 
information.  The  resulting,  and  typically  incomplete,  figure  is  fed  back  to  the  "early 
vision"  stage  for  long-range  completion  via  filling-in.  The  reconstructed  image  is  then  re¬ 
presented  to  the  recognition  system  for  global  functions  such  as  object  recognition.  In 
the  CONFIGR  algorithm,  the  smallest  independent  image  unit  is  the  visible  pixel,  whose 
size  defines  a  computational  spatial  scale.  Once  pixel  size  is  fixed,  the  entire  algorithm 
is  fully  determined,  with  no  additional  parameter  choices.  Multi-scale  simulations 
illustrate  the  vision/recognition  system.  Open-source  CONFIGR  code  is  available  online, 
but  all  examples  can  be  derived  analytically,  and  the  design  principles  applied  at  each 
step  are  transparent.  The  model  balances  filling-in  as  figure  against  complementary 
filling-in  as  ground,  which  blocks  spurious  figure  completions.  Lobe  computations  occur 
on  a  subpixel  spatial  scale.  Originally  designed  to  fill-in  missing  contours  in  an 
incomplete  image  such  as  a  dashed  line,  the  same  CONFIGR  system  connects  and 
segments  sparse  dots,  and  unifies  occluded  objects  from  pieces  locally  identified  as 
figure  in  the  initial  recognition  stage.  The  model  self-scales  its  completion  distances, 
filling-in  across  gaps  of  any  length,  where  unimpeded,  while  limiting  connections  among 
dense  image-figure  pixel  groups  that  already  have  intrinsic  form.  Long-range  image 
completion  promises  to  play  an  important  role  in  adaptive  processors  that  reconstruct 
images  from  highly  compressed  video  and  still  camera  images. 

Laminar  cortical  dynamics  of  Cognitive  and  motor  working  memory,  sequence 
learning  and  performance:  After  the  brain  perceives  and  recognizes  individual  objects 
and  events,  it  needs  to  organize  them  into  sequential  plans  in  order  to  generate 
situationally-appropriate  actions.  In  order  to  do  this,  it  needs  to  store  sequences  of 
events  temporarily  in  a  working  memory.  How  does  the  brain  carry  out  working  memory 
storage,  categorization,  and  voluntary  performance  of  event  sequences?  The  LIST 


PARSE  neural  model  was  developed  by  Grossberg  and  Pearson  to  propose  an  answer 
to  this  question  that  unifies  the  explanation  of  cognitive,  neurophysiological,  and 
anatomical  data  from  humans  and  monkeys.  It  quantitatively  simulates  human  cognitive 
data  about  immediate  serial  recall  and  free  recall,  and  monkey  neurophysiological  data 
from  the  prefrontal  cortex  obtained  during  sequential  sensory-motor  imitation  and 
planned  performance.  The  model  clarifies  why  both  spatial  and  non-spatial  working 
memories  share  the  same  type  of  circuit  design.  It  proposes  how  the  laminar  circuits  of 
lateral  prefrontal  cortex  carry  out  working  memory  storage  of  event  sequences  within 
layers  6  and  4,  how  these  event  sequences  are  unitized  through  learning  into  list 
chunks  within  layer  2/3,  and  how  these  stored  sequences  can  be  recalled  at  variable 
rates  that  are  under  volitional  control  by  the  basal  ganglia.  These  laminar  prefrontal 
circuits  are  variations  of  laminar  circuits  in  the  visual  cortex  that  have  been  used  to 
explain  data  about  how  the  brain  sees.  These  examples  from  visual  and  prefrontal 
cortex  illustrate  how  laminar  neocortex  can  represent  both  spatial  and  temporal 
information,  and  open  the  way  towards  understanding  how  other  behaviors  may  be 
represented  and  controlled  by  variations  on  a  shared  laminar  neocortical  design.  When 
enough  examples  of  such  laminar  computing  are  developed,  they  will  open  the  way 
towards  designing  families  of  VLSI  chips,  all  variations  of  a  shared  laminar  cortical 
design,  that  can  be  self-consistently  integrated  into  an  autonomous  controller  of  multiple 
modalities  of  intelligence. 

Coordinating  saccadic  and  smooth  pursuit  eye  movements  during  visual  tracking 
of  unpredictably  moving  targets:  As  illustrated  by  the  above  projects  on  visual 
learning  and  recognition,  oculomotor  tracking  of  moving  objects  is  an  important 
component  of  visually  based  cognition,  planning,  and  decision-making.  The  ARTSCAN 
and  model,  in  particular,  demonstrates  how  eye  movements  may  be  used  to  learn  view- 
invariant  object  categories,  but  it  does  not  explain  how  the  brain  generates  these  eye 
movements.  The  brain  has  a  fovea  with  high  visual  acuity  that  must  be  moved  efficiently 
across  a  scene  to  see  and  understand  it  well.  Ballistic  or  saccadic  eye  movements,  by 
themselves,  would  greatly  diminish  the  amount  of  time  that  the  fovea  fixates  objects  of 
interest.  The  brain  intelligently  coordinates  saccades  with  predictive  smooth  pursuit 
movements  to  maximize  the  amount  of  time  that  a  moving  target  is  foveated.  In 
particular,  the  saccadic  and  smooth  pursuit  systems  interact  to  often  choose  the  same 
target,  and  to  maximize  its  visibility  through  time.  How  does  the  brain  coordinate  these 
two  types  of  eye  movements  to  track  objects  that  move  in  unpredictable  directions  and 
speeds?  How  do  multiple  brain  regions  interact,  including  frontal  cortical  areas,  to 
decide  the  choice  of  a  target  among  several  competing  moving  stimuli?  How  can  these 
insights  be  used  to  develop  more  effective  machine  tracking  methods:  Saccadic  eye 
movements  rapidly  foveate  peripheral  visual  or  auditory  targets,  and  smooth  pursuit  eye 
movements  keep  the  fovea  pointed  toward  an  attended  moving  target.  Analyses  of 
tracking  data  in  monkeys  and  humans  reveal  systematic  deviations  from  predictions  of 
the  simplest  model  of  saccade-pursuit  interactions,  which  would  use  no  interactions 
other  than  common  target  selection  and  recruitment  of  shared  motoneurons.  Instead, 
saccadic  and  smooth  pursuit  movements  cooperate  to  cancel  errors  of  gaze  position 
and  velocity,  and  thus  to  maximize  target  visibility  through  time.  Moreover,  saccades  are 
calibrated  to  correctly  foveate  a  target  despite  its  continued  motion  during  the  saccade? 


A  neural  model  has  been  developed  by  Bullock,  Grossberg,  and  Srihasam  to  provide 
answers  to  such  questions.  The  modeled  interactions  encompass  motion  processing 
areas  MT,  MST,  FPA,  DLPN  and  NRTP;  saccade  planning  and  execution  areas  FEF 
and  SC;  the  saccadic  generator  in  the  brain  stem;  and  the  cerebellum.  Simulations 
illustrate  the  model’s  ability  to  functionally  explain  and  quantitatively  simulate 
anatomical,  neurophysiological  and  behavioral  data  about  saccade-pursuit  target 
tracking. 

Visually-guided  steering,  obstacle  avoidance,  and  route  selection:  Tracking  of 
moving  targets  often  occurs  while  a  human,  or  mobile  robot,  navigates  an  environment. 
How  is  optic  flow  information  used  to  steer  towards  a  goal  while  avoiding  obstacles?  A 
Steering,  Tracking,  And  Route  Selection  (STARS)  neural  model  was  developed  by 
Elder,  Grossberg,  and  Mingolla  to  explain  how  humans  can  approach  a  goal  object  on 
foot  while  steering  around  obstacles  to  avoid  collisions  in  a  cluttered  environment.  The 
model  uses  optic  flow  from  a  3D  virtual  reality  environment  to  determine  the  position  of 
objects  based  on  motion  discontinuities,  and  computes  heading  direction,  or  the 
direction  of  self-motion,  from  global  optic  flow.  The  cortical  representation  of  heading 
interacts  with  the  representations  of  a  goal  and  obstacles  in  such  a  way  that  the  goal 
acts  as  an  attractor  of  heading,  while  obstacles  act  as  repellers.  This  result  clarifies 
behavioral  data  that  demonstrate  such  an  attractor/repeller  scheme.  In  addition,  the 
model  maintains  fixation  on  the  goal  object  by  generating  smooth  pursuit  eye 
movements,  whose  finer  control  is  clarified  by  the  model  of  Bullock,  Grossberg,  and 
Srihasam.  Eye  rotations  can  distort  the  optic  flow  field,  complicating  heading 
perception,  and  the  model  uses  extraretinal  signals  to  correct  for  this  distortion  and 
accurately  represent  heading.  The  model  explains  how  motion  processing  mechanisms 
in  cortical  areas  MT,  MST,  and  VIP  can  be  used  to  guide  steering.  The  model 
quantitatively  simulates  human  psychophysical  data  about  visually-guided  steering, 
obstacle  avoidance,  and  route  selection.  The  model  architecture  captures  the 
attractor/repeller  dynamics  of  steering  behavior,  and  clarifies  how  heading  and  eye 
movements  work  together  during  complex  steering  tasks. 

A  modular  architecture  for  implementing  cortical  processing  through  forward, 
feedback,  and  lateral  connections:  A  CMOS  cortical  processor  chip  has  been 
fabricated  in  MIT  Lincoln  Labs  3D  0.18  um  CMOS  technology  for  a  64  x  64  processor 
that  incorporates  1  million  transistors  in  1.5  x  1.5  mm  of  silicon.  10  3D  CMOS  dies  were 
received  in  May  2006.  A  printed  circuit  board  has  been  designed  to  test  the  chips. 
Preliminary  results  are  encouraging. 
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•  Carpenter,  G.A.  &  Grossberg,  S.,  U.S.  Patent  No.  5,142,590:  Pattern 
recognition  system  (Filed:  November  27,  1985.  Issued:  August  25,  1992. 
European  Patent  No.  0244483,  issued  July  15,  1992).  (ART  1). 

•  Carpenter,  G.A.  &  Grossberg,  S.,  U.S.  Patent  Nos.  4,914,708  and 
5,133,021:  System  for  self  organization  of  stable  category  recognition 
codes  for  analog  patterns  (Filed:  June  19,  1987.  Issued:  April  3,  1990  and 
July  21,  1992).  (ART  2). 

•  Carpenter,  G.A.  &  Grossberg,  S.,  U.S.  Patent  No.  5,311,601:  Pattern 
recognition  system  with  variable  selection  weights  (Filed:  January  12, 
1990.  Issued:  May  10,  1994).  (ART  3). 

•  Carpenter,  G.A.,  Grossberg,  S.,  &  Rosen,  D.B.,  U.S.  Patent  No. 


5,157,738:  Rapid  category  learning  and  recognition  system  (Filed: 
December  19,  1990.  Issued:  October  20,  1992).  (ART  2  A). 

•  Carpenter,  G.A.,  Grossberg,  S.,  &  Reynolds,  J.H.,  U.S.  Patent  No. 
5,214,715:  Predictive  self  organizing  neural  network.  (Filed:  January  31, 
1991.  Issued:  May  25,  1993).  (ARTMAP). 

14.  Grossberg,  S.,  Founding  member  of  the  International  Neural  Network  Society 
(INNS)  Governing  Board  (1987-present). 

15.  Grossberg,  S.,  Founding  Editor-in-Chief  of  Neural  Networks,  the  official  journal  of 
the  International  Neural  Network  Society  (INNS),  the  European  Neural  Network 
Society  (ENNS),  and  the  Japanese  Neural  Network  Society  (JNNS).  Re-elected 
by  INNS  Board  of  Governors  for  each  five-year  term,  1987-present. 

16.  Grossberg,  S.,  Listed  in  Highly  Cited  Researchers  database  by  the  ISI  Web  of 
Science  (less  than  1/2  %  of  all  published  researchers). 

17.  Grossberg,  S.,  Editor  of  the  journals  Adaptive  Behavior,  Applied  Intelligence-, 
Behavioral  and  Brain  Sciences  (Associate  Editor  for  Computational 
Neuroscience);  Behavioural  Processes',  Cognition  and  Brain  Theory,  Cognitive 
Brain  Research ;  Cognitive  Processing-,  Cognitive  Science ;  IEEE  Expert ;  IEEE 
Transactions  on  Neural  Networks-,  Information  Science-,  International  Journal  of 
Cognitive  Science ;  International  Journal  of  Uncertainty,  Fuzziness,  and 
Knowledge-Based  Systems ;  Journal  of  Cognitive  Neuroscience-,  Journal  of 
Mathematical  Psychology,  Journal  of  Theoretical  Neurobiology,  Mathematical 
Biosciences ;  Mind  and  Society,  Neural  Computation-,  Nonlinear  Analysis. 

18.  Grossberg,  S.,  General  Chairman  and  Organizer,  Fifth  International  Conference 
on  Cognitive  and  Neural  Systems  (ICCNS),  Boston  University,  May  2001. 

19.  Grossberg,  S.,  Honorary  Chairman,  International  Conference  on  Computational 
Intelligence  for  Modeling,  Control,  and  Automation,  Las  Vegas  NV,  July  2001. 

20.  Grossberg,  S.,  Organizing  Committee  member,  annual  Artificial  Neural  Networks 
in  Engineering  (ANNIE)  conference,  St.  Louis  MO,  November  2001. 

21.  Grossberg,  S.,  International  Advisory  Board,  Mind  and  Society  (new  journal), 
February  2002. 

22.  Grossberg,  S.,  Advisory  Board,  Advanced  Information  Processing,  new  book 
series,  Springer-Verlag  (Computer  Science),  February  2002. 

23.  Grossberg,  S.,  Editorial  Board,  Mathematical  Modeling:  Theory  and  Applications 
(monographs),  Kluwer,  February  2002. 

24.  Grossberg,  S.,  Charles  River  Award,  Behavioral  Toxicology  Society,  April  2002. 

25.  Grossberg,  S.,  IEEE  Senior  Member,  May  2002. 

26.  Grossberg,  S.,  General  Chairman  and  Organizer,  Sixth  International  Conference 
on  Cognitive  and  Neural  Systems  (ICCNS),  Boston  University,  May  2002. 

27.  Grossberg,  S.,  Fellow,  American  Psychological  Society  (APS),  June  2002. 

28.  Grossberg,  S.,  International  Program  Committee  for  the  second  Euro- 
International  Symposium  on  Computational  Intelligence  (ISCI),  Kosice,  Slovakia, 
June  2002. 

29.  Grossberg,  S.,  Program  Committee  member,  International  Conference  on 
Information  Processing  and  Management  of  Uncertainty  in  Knowledge-Based 
Systems,  Annecy,  France,  2002. 


30.  Grossberg,  S.,  Advisory  Committee  for  IEEE  International  Conference  on 

Industrial  Technology  (also  invited  to  be  Keynote  Speaker;  declined),  Bangkok, 
Thailand,  2002. 

31.  Grossberg,  S.,  Honorary  Program  Committee  member,  Seventh  Brazilian 

Symposium  on  Artificial  Neural  Networks,  2002. 

32.  Grossberg,  S.,  International  Advisory  Board,  Encyclopedia  of  Consciousness 
and  Subjectivity:  Phenomenology,  Cognitive  Neuroscience,  and  Psychiatry, 
Kluwer  Press,  2002. 

33.  Grossberg,  S.,  Editor  of  a  new  journal  on  Current  Opinion  in  Cognitive 

Neurodynamics,  2005. 

34.  Grossberg,  S.,  Editor  of  the  new  International  Journal  of  Humanoid  Robotics, 
2005. 

35.  Grossberg,  S.,  Editor  of  the  new  International  Journal  of  Hybrid  Intelligent 

Systems,  2005. 

36.  Grossberg,  S.,  One  of  five  founding  editors  of  Brains,  Minds  &  Media,  a  new  e- 
journal  devoted  to  educational  technology  and  teaching  in  the  neural  and 
cognitive  sciences,  2005. 

37.  Grossberg,  S.,  Editorial  board  member  of  the  journal  Advanced  Information  and 
Knowledge  Processing,  2005. 

38.  Grossberg,  S.,  Competitively  chosen  two-hour  tutorial  with  Ennio  Mingolla 

entitled  “How  the  Brain  Sees”  at  the  first-ever  annual  Vision  Sciences  Society 
(VSS)  Satellite  Workshop,  May  2005.  This  was  the  only  modeling  presentation 
selected. 

39.  Grossberg,  S.,  General  Chairman  of  the  Ninth  International  Conference  on 
Cognitive  and  Neural  Systems  (ICCNS),  Boston  University,  May  2005. 

40.  Grossberg,  S.:  Program  Committee  member  for  the  conference  on  Information 
Processing  and  Management  of  Uncertainty  in  Knowledge-Based  Systems, 
2005. 

41.  Grossberg,  S.,  Invited  to  be  an  editor  of  the  International  Journal  of  Neural 
Systems,  July  2005. 

42.  Grossberg,  S.,  Honored  at  a  two-day  meeting  “An  Anniversary  Conference 
Celebrating  Steve  Grossberg@65  and  CNS@15”,  Boston  University,  September 
2005  (http://www.cns. bu.edu/events/SeDt2005conference/index.html). 

43.  Grossberg,  S.,  Invited  to  be  an  editor  of  the  journal  Cognitive  Neurodynamics, 
February  2006. 

44.  Grossberg,  S.,  Elected  IEEE  Fellow,  November  2005. 

45.  Mingolla,  E.,  Editorial  board  member  of  the  journals  Neural  Networks  and 
Ecological  Psychology. 

46.  Mingolla,  E.,  Guest  Editor  of  the  journal  Spatial  Vision,  Volume  18,  No.  2,  2005. 

47.  Mingolla,  E.,  Competitively  chosen  two-hour  tutorial  with  Stephen  Grossberg 
entitled  “How  the  Brain  Sees”  at  the  first-ever  annual  Vision  Sciences  Society 
(VSS)  Satellite  Workshop,  May  2005.  This  was  the  only  modeling  presentation 
selected. 

48.  Mingolla,  E.,  Helmholtz  Award,  International  Neural  Network  Society  (INNS), 
2007. 

49.  Schwartz,  E.,  Editorial  board  member  of  the  journal  Neural  Networks. 


Patent  Activity 

No  patents  were  submitted  or  issued  during  the  award  period. 


Technology  Transfer 

HRL  Laboratories:  HRL  (formerly  Hughes  Research  Laboratories)  is  now  owned  by 
Boeing,  GM,  and  Raytheon.  Professors  Carpenter,  Grossberg,  and  Mingolla  worked 
with  HRL  researchers  to  develop  a  large-scale  system  to  control  an  autonomous 
intelligent  agent.  DARPA  funds,  from  the  BICA  (Biologically  Inspired  Cognitive 
Architectures)  program,  have  been  funding  this  collaborative  relationship;  see 
http://www.hrl.com/html/prsRls  060501  .htm.  Many  of  these  designs  grew  out  of  ONR 
MURI  funding.  CNS  faculty  have  consulted  with  HRL  scientists,  but  this  limited 
relationship  played  a  key  role  in  getting  HRL  a  Phase  I  award.  CNS  will  work  with  HRL 
to  try  to  get  a  Phase  II  BICA  grant. 

Nanyang  Technological  University:  Faculty,  notably  Professors  Ah-Hwee  Tan  and  Alex 
Tay,  at  this  University  have  made  plans  to  incorporate  CNS  algorithms  into  adaptive 
autonomous  mobile  robots  for  industry.  Professor  Tay  visited  CNS  in  2005  and 
Professor  Tan  visited  in  2006  to  discuss  possible  approaches  with  Professors  Carpenter 
and  Grossberg.  A  colleague  of  Professors  Tan  and  Tay,  Dr.  Gee  Wah  Ng,  Head  of  the 
Advanced  Analysis  and  Fusion  Laboratory  of  the  DSO  National  Laboratories  in 
Singapore,  visited  CNS  on  sabbatical  in  2006-2007  to  continue  this  collaboration. 

Intellectual  Ventures,  LLC  (http://www.intellectualventures.com/):  Sale  by  Boston 
University  of  five  ART  patents  (November,  2005).  Exclusive  worldwide  license  sold  by 
Boston  University  to  Intellectual  Ventures,  LLC.  See  the  Honors  and  Awards  section  of 
this  report  for  further  details. 

MIT  Lincoln  Laboratory:  MURI  vision,  recognition,  and  control  neural  algorithms  have 
been  transferred  to  MIT  Lincoln  Laboratory  for  use  in  cutting-edge  DoD  applications, 
and  transferred  from  there  to  other  DoD  labs.  More  than  10  CNS  PhDs  were  hired  by 
Lincoln  to  facilitate  technology  transfer.  That  cooperation  ended  when  its  leader  left 
Lincoln  to  work  on  national  security  issues  in  a  company.  It  is  now  being  rekindled  by 
one  CNS  PhD,  Dr.  William  Ross,  who  was  hired  by  Lincoln  Lab.  Ross,  now  a  Lincoln 
assistant  group  leader,  has  arranged  for  an  employee  to  get  Lincoln  funding  to  earn  an 
MA  degree  at  CNS  and  to  do  collaborative  research  between  CNS  and  Lincoln  Lab. 
More  cooperation  is  anticipated  in  the  future. 

Technology  Website:  The  CNS  Technology  Lab  under  Professor  Carpenter’s  direction 
has  developed  a  website  (http://cns.bu.edu/techlab)  to  present  open  source  code, 
challenge  problems,  recent  image  processing  and  adaptive  pattern  recognition  and 
prediction  algorithms  for  technology,  data  sets,  user-friendly  biological  neural  models, 
benchmark  studies,  tutorials,  and  current  research  articles  for  use  by  the  world-wide 
technology  community.  This  resource  will  create  many  new  technology  transfer 
opportunities  and  applications.  Many  companies  have  used  CNS  image-processing  and 
pattern  recognition  algorithms  in  their  products.  The  website  will  expedite 


commercialization  by  making  documented  CNS  algorithms  more  readily  available. 

VLSI  Development:  The  neuromorphic  circuits  and  architectures  resulting  from  the 
research  under  the  MURI  award  have  been  implemented  in  the  MIT  Lincoln  Lab 
(MIT/LL)  3  tier  FDSOI  3D  CMOS  technology. 

Leveraging  ONR  support  into  other  grants:  The  NGA  and  NSF  CELEST  grants  were 
funded  based  on  algorithms  developed  in  significant  measure  under  the  MURI  award. 


ONR  DATABASE  STATISTICS 


Postdoctoral  Information 

0  Postdoc  Minority  Women 

0  Postdoc  Non-Minority  Women 

0  Postdoc  Minority  Men 

0  Postdoc  Non-Minority  Men 


PI/Co-PI  Information 

0  PI/Co-PI  Minority  Women 

0  PI/Co-PI  Non-Minority  Women 

0  PI/Co-PI  Minority  Men 

1  PI/Co-PI  Non-Minority  Men 


Graduate  Student  Information 

0  Graduate  Minority  Women 

5  Graduate  Non-Minority  Women 

2  Graduate  Minority  Men 

44  Graduate  Non-Minority  Men 

Undergraduate  Student  Information 

0  Undergrad  Minority  Women 

0  Undergrad  Non-Minority  Women 

0  Undergrad  Minority  Men 

0  Undergrad  Non-Minority  Men 


Under-represented  or  minority  groups  include  Blacks,  Hispanics,  and  Native  Americans.  Asians  are  not 
considered  an  under-represented  or  minority  group  in  science  and  engineering. 

Supported  at  least  25%  per  year  on  contract/grant. 

Publication  Totals 

65  Peer-Reviewed  Journal  Articles 

1 5  Books  or  Book  Chapters 

1 1  Technical  Reports 

129  Conference  Abstracts  and  Proceedings 

0  Patents  Issued 

0  Patents  Pending 


Human  Subjects  NO 

Animal  Subjects  NO 

Recombinant  DNA  NO 

PhD  Degrees  Granted  39 

Honors  and  Awards  49 
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